Through digitization of image devices and networks, it is now possible to connect any image devices to one another, increasing the degree of freedom in exchanging images. There has been prepared an environment where a user can freely handle images without being restricted by differences between systems. For example, a user now can output an image taken by a digital still camera to a printer, have it open to the public on a network, or view it on a household television set.
However, each system now needs to be compatible with various image formats, and there is naturally a need for improvements to image format conversions. For example, image size conversions are done frequently, and there is needed an up-converter (a conversion device for increasing the number of pixels, the number of lines, etc.). For example, printing on A4 paper (297 mm×210 mm) with a resolution of 600 dpi requires an original of 7128 pixels×5040 lines, but the resolution of most digital still cameras is lower than this, thus requiring an up-converter. For an image being open to the public on a network, the ultimate form of output has not been determined, whereby such an image needs to be converted to an appropriate image size each time an output device is determined. For household television sets, digital terrestrial television services have been started, and conventional standard television images and HD (High Definition) television images coexist, whereby image size conversions are used frequently.
Conventional image enlargement processes are based on the difference in luminance value between pixels. That is, for image enlargement, luminance values are interpolated by a bilinear method or a bicubic method in order to newly produce pixel data that were not present at the time of sampling (see, Non-Patent Document 1). With interpolation, it is only possible to produce intermediate values between sampled data, and the sharpness of an edge, or the like, tends to deteriorate. In view of this, techniques have been disclosed in the art in which an interpolated image is used as an initial enlarged image, after which edge portions are extracted and the edges are selectively enhanced (see Non-Patent Document 2 and Non-Patent Document 3). Particularly, Non-Patent Document 3 provides an improvement by incorporating the multiresolution representation and the Lipschitz index, whereby the edge enhancement is done selectively according to the sharpness of the edge.
However, if an image enlarging conversion is performed by using the difference in luminance between pixels, it is difficult to separate between an edge component and noise, and it is likely that an image conversion deteriorates the sharpness of an edge or the texture (the pattern appearing on the surface of an object). Specifically, in image enlargement, the process enhances edge portions of an initial enlarged image, which have been blurred by interpolation, whereby it may also enhance noise along with edge portions, thus deteriorating the image quality. Moreover, the edge enhancement of an interpolated image in an image enlargement is an empirical approach with no explicit countermeasures against noise, and it is therefore not possible to guarantee the image quality after image conversion.
In order to solve the problem set forth above, a technique has been disclosed in the art in which the texture of an input image is analyzed so as to produce a spatially super-resolution image by using the analyzed texture feature quantity and a super-resolution texture feature quantity obtained from a super-resolution image having a higher spatial resolution (see Patent Document 1). The texture feature quantity is produced from the spatial frequency response quantity obtained by a Fourier transformation, a wavelet transformation, or the like, and the texture of the input image is determined based on a feature quantity histogram formed by a combination of the resolution and the material. Then, a super-resolution process is performed on the texture feature quantity by using a texture feature quantity vector conversion table to thereby produce a super-resolution texture feature quantity vector. The super-resolution texture feature quantity vector is converted to a luminance image through an inverse conversion of the conversion process performed on the spatial frequency response for producing the texture feature quantity from an input image, thus obtaining a resolution-increased output image. As described above, the texture feature quantity is subjected to the super-resolution process for each material, thereby enabling an image enlargement without deteriorating the feel of the material texture, thus providing an effect that there is no image quality deterioration.
However, the feature quantity histogram and the texture feature quantity vector conversion table need to be stored as a database. For example, when the condition of the lighting illuminating the object changes, the luminance value of each pixel changes, and thus the texture feature quantity changes. Therefore, it will be necessary to produce, and add to the database, the feature quantity histogram and the texture feature quantity vector conversion table each time as necessary.
In order to solve the problem set forth above, it is effective to decompose the luminance Iv captured by a camera CAM into geometric characteristics and optical characteristics of an object OBJ and a visual environment VE to classify the parameters into resolution-independent parameters and resolution-dependent parameters, as shown in FIG. 39, wherein the feature quantity histogram and the texture feature quantity vector conversion table are produced only for the resolution-dependent parameters. Thus, even if the resolution-independent parameters vary, it is not necessary to add the feature quantity histogram and the texture feature quantity vector conversion table. Consider a case where the luminance Iv captured by the camera CAM is given by an illumination equation of (Expression 1).
                    [                  Formula          ⁢                                          ⁢          1                ]                                                                                                                I                v                            =                                                I                  a                                +                                  I                  d                                +                                  I                  s                                                                                                        =                                                I                  a                                +                                                      E                    i                                    ⁡                                      (                                                                                            k                          d                                                ⁢                                                  ρ                          d                                                                    +                                                                        k                          s                                                ⁢                                                  ρ                          s                                                                                      )                                                                                                          (                  Expression          ⁢                                          ⁢          1                )            
Herein, Ia is the luminance of the ambient light, Id is the luminance of the diffuse reflection component, Is is the luminance of the specular reflection component, Ei is the illuminance at the point of interest P, the vector N is the surface normal vector, the vector L is the light source vector representing the direction of the light source IL, ρd is the diffuse reflection component bidirectional reflectance, ρs is the specular reflection component bidirectional reflectance, kd is the diffuse reflection component ratio, and ks is specular reflection component ratio, wherein kd+ks=1. Moreover, the specular reflection component bidirectional reflectance vector ρs is given by (Expression 2), for example, and is decomposed into a plurality of geometric parameters and optical parameters.
                    [                  Formula          ⁢                                          ⁢          2                ]                                                                                  ρ            s                    =                                                    F                λ                            π                        ⁢                          DG                                                (                                                            N                      _                                        ·                                          V                      _                                                        )                                ⁢                                  (                                                            N                      _                                        ·                                          L                      _                                                        )                                                                    ⁢                                  ⁢                  D          =                                    1                              4                ⁢                                                                  ⁢                                  m                  2                                ⁢                                  cos                  4                                ⁢                β                                      ⁢                          ⅇ                              -                                                      [                                                                  (                                                  tan                          ⁢                                                                                                          ⁢                          β                                                )                                            /                      m                                        ]                                    2                                                                    ⁢                                  ⁢                  G          =                      min            ⁢                          {                              1                ,                                                      2                    ⁢                                          (                                                                        N                          _                                                ·                                                  H                          _                                                                    )                                        ⁢                                          (                                                                        N                          _                                                ·                                                  V                          _                                                                    )                                                                            (                                                                  V                        _                                            ·                                              H                        _                                                              )                                                  ,                                                      2                    ⁢                                          (                                                                        N                          _                                                ·                                                  H                          _                                                                    )                                        ⁢                                          (                                                                        N                          _                                                ·                                                  L                          _                                                                    )                                                                            (                                                                  V                        _                                            ·                                              H                        _                                                              )                                                              }                                      ⁢                                  ⁢                              F            λ                    =                                    1              2                        ⁢                                                            (                                      g                    -                    c                                    )                                2                                                              (                                      g                    +                    c                                    )                                2                                      ⁢                          (                              1                +                                                                            [                                                                        c                          ⁡                                                      (                                                          g                              +                              c                                                        )                                                                          -                        1                                            ]                                        2                                                                              [                                                                        c                          ⁡                                                      (                                                          g                              -                              c                                                        )                                                                          +                        1                                            ]                                        2                                                              )                                      ⁢                                  ⁢                              g            2                    =                                    n              2                        +                          c              2                        -            1                          ⁢                                  ⁢                  c          =                      (                                          L                _                            ·                              H                _                                      )                                              (                  Expression          ⁢                                          ⁢          2                )            
Herein, H is the middle vector between the viewpoint vector V and the lighting vector L, β represents the angle between the middle vector H and the surface normal vector N. Herein, m is a coefficient representing the roughness of the object surface, wherein when m is small, a strong reflection is exhibited in an area where the angle β is small, i.e., in the vicinity of the surface normal vector N, and when m is large, the reflection distribution expands to an area where the angle β is large, i.e., an area distant from the surface normal vector N. G is the geometric attenuation factor, and represents the influence of shading due to irregularities on the substrate surface. Herein, n is the refractive index.
Among all the illumination equation parameters of (Expression 1) and (Expression 2), the light source vector L, the viewpoint vector V and the illuminance Ei of the lighting are those dependent on the visual environment VE, and are constant or vary smoothly across all the pixels, thus being highly dependent of the image resolution. The diffuse reflection component ratio kd, the specular reflection component ratio ks, the object surface roughness m and the refractive index n are those dependent on the material of the object OBJ, and are constant or vary smoothly across all the pixels within the same material, thus being highly independent of the image resolution. The surface normal vector N and the diffuse reflection component bidirectional reflectance pd are not bound by the visual environment VE and the material of the object OBJ, and may take various parameter values for each pixel, thus being highly dependent on the image resolution.
Thus, super-resolution processes performed on different parameter images are classified into three as shown in FIG. 40. First, since the light source vector L, the viewpoint vector V and the illuminance Ei of the lighting are constant or vary smoothly across all the pixels and are highly independent of the image resolution, the low-resolution parameter images LL, VL and (Ei)L can be converted to the high-resolution parameter images LH, VH and (Ei)H by copying the pixel values of the low-resolution images or interpolating the pixel values from adjacent pixels. Second, since the diffuse reflection component ratio kd, the specular reflection component ratio ks, the object surface roughness m and the refractive index n are constant or vary smoothly across all the pixels within the same material and are highly independent of the image resolution, the low-resolution parameter images ksL, mL and nL can be converted to the high-resolution parameter images ksH, mH and nH by determining the material through texture analysis sections TA1 to TA3 and copying the pixel values of the low-resolution images or interpolating the pixel values from adjacent pixels for each material. Third, since the surface normal vector N and the diffuse reflection component bidirectional reflectance ρd are highly dependent on the image resolution, the super-resolution process is performed by texture feature quantity vector conversion through texture super-resolution sections TSR1 and TSR2. Therefore, the resolution and the material of the low-resolution parameter images NL and ρdL are determined based on a feature quantity histogram through texture analysis sections TA4 and TA5, and the texture feature quantity vector conversion is performed by the texture super-resolution section TSR1 and TSR2 to obtain the high-resolution parameter images NH and ρdH.
As described above, the texture feature quantity is subjected to the super-resolution process for each material, thereby enabling an image enlargement without deteriorating the feel of the material texture, thus providing an effect that there is no image quality deterioration.
Non-Patent Document 1: Shinji Araya, “Meikai 3-Jigen Computer Graphics (3D Computer Graphics Elucidated)”, Kyoritsu Shuppan Co., Ltd., pp. 144-145, Sep. 25, 2003
Non-Patent Document 2: H. Greenspan, C. H. Anderson, “Image enhancement by non-linear extrapolation in frequency space”, SPIE Vol. 2182, Image and Video Processing II, 1994
Non-Patent Document 3: Makoto Nakashizuka, et al., “Image Resolution Enhancement On Multiscale Gradient Planes”, Journal of the Institute of Electronics, Information and Communication Engineers D-II Vol. J81-D-II, No. 10, pp. 2249-2258, October 1998
Non-Patent Document 4: Image Processing Handbook Editing Committee ed., “Image Processing Handbook”, Shokodo Co., Ltd., pp. 393, Jun. 8, 1987
Non-Patent Document 5: Norihiro Tanaka, Shoji Tominaga, “A Method For Estimating Refractive Index Of inhomogeneous Material from Images”, Technical Report of the Institute of Electronics, Information and Communication Engineers, PRMU2002-175, pp. 37-42 January 2003
Patent Document 1: International Publication WO2005/067294 (FIG. 11)